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Abstract: This paper explores evaluation strategies to gauge the impact of a novel instructional design on international 
community participation online. This is done by conceptualizing and devising indicators for measuring “engagement” 
online amongst marginalized adult communities worldwide. In doing so, a review of online evaluation literature is 
conducted. In comparing dialogue sessions based on an ongoing traditional model to the new instructional approach, 
various challenges are faced in “measuring” asynchronous discussion. While the initial findings of marginal increase in 
engagement with the adapted instructional approach is not sufficient to prove that the new model works, this paper 
demonstrates various strategies/ challenges in evaluating dialectic engagement. 
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1. Introduction 

In the last two decades, the subject of computer-mediated collaborative environments has gained attention 
as more individuals across the globe seek to express themselves and understand each other through 
diverse online avenues. Much has been made about “virtual communities” that shape themselves along 
weak social network ties, celebrating these novel online engagements and connectedness (Rheingold, 
1993). Of course these celebrations are countered with equally vehement derisions and cautions of loss of 
real time and face-to-face relationships, the (de)socializations of what is seen as a new virtual public 
(Lipnack & Stamps, 1997). Be it celebration, caution, derision or processes of “figuring out,” there is no doubt 
however that online social networks are here to stay and more importantly, expand in often unpredictable 
ways. 

Much has been researched in areas of online identity in contestation to socially constructed stereotypes 
(Turkle, 1995; Wilber, 1997; Kendell, 1998; Kollock, 1999; Smith, 1992) as has the role of computer- 
mediated communications (CMC) in the complex interplay of text, talk and learning (Warschauer, 1997) and 
membership through social learning amongst communities online (Kim, 2000). As online communities get 
more competitive and formalized in their facilitation and mediation, strategies are sought to enhance the 
participant’s online experiences and engagements. There is much literature on strategies to moderate formal 
online distance learning (Levine & Sun, 2002; Discenza, Schenk & Howard, 2002; Assie-Lumumba, 2004), 
metacognitive or higher-order thinking strategies to enhance such learning (King, 1992; Schwartz, Bransford, 
& Sears, 2005) and the use of computers as a tool to advance these strategies (Scardamalia & Bereiter, 
1991; Pea, 1994; Lin, Hmelo, Kinzer & Secules, 1999). 

Further, evaluation of such strategies is now a pursued field as it is looked upon as essential to the 
understanding and proliferating of conducive behaviors, online and offline (Kendall, 1999; Leander & McKim, 
2003). However, there is limited research conducted on strategies employed to facilitate asynchronous 
online dialogue forums amongst marginalized adults across socio-cultural and geographic environments. In 
particular, instructional strategies and evaluative frameworks to guide online dialogue amongst 
“disadvantaged” groups are few and far between. Therefore, this paper explores the potential of an 
instructional design strategy amongst such participants through demonstrated evaluative measures. 
Specifically, the scope of this paper is to gain some insight to the following questions through specific 
evaluation: 

1. Is there a change in “engagement” with a change in the instructional design model online? 

2. Does the new (CEBLE) instructional design model enhance “engagement” within an online dialogue 
setting? 

This paper contains three sections. The first section surveys the understandings of online community 
participation and its evaluation strategies. The second section contains the details of the pilot project. This 
entails a description of the CEBLE model and the pilot setting, research goals, comparative results of the 
traditional to the applied CEBLE model and an analysis of findings. The third section concludes by stating 
challenges, inferences, and avenues for further research. 
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2. Understanding online community participation 

One of the main challenges in the understanding of collaborative online discourse is in its transient nature. 
Nentwich (2004) states that to capture the good ideas that come up within a group, it is useful to have an 
archives feature. Also, in describing group dynamics, he states that online communities can serve as a portal 
for interactive interaction, information dissemination and exchange, capable of producing a synergistic 
relationship among the members of the group. Hine (2000) likens such interaction to a “flow” that achieves 
“relative spatiality across disparate resources” within social-material networks (p.76). The nature of the “flow” 
as Kendall (2004) professes is influenced more by asynchronous than synchronous factors, a form of 
communication that involves an interval between the sender and receiver of messages. This is corroborated 
by Moss and Shank’s (2002) findings on asynchronous discussions on Bulletin Board Systems (BBS). Based 
on their findings, they inferred that asynchronous discussion could produce higher participation & quality 
discussion as compared to synchronous discussion (p.24). Also, they found that participants collaborated 
with more equity using BBS. Kendall (2004) warns us though that in evaluating socially disparate sub-cultural 
group participants, we should not presume that all online interaction occurs within the same homogenous 
cultural context separate from the different cultures of the participants who contribute to it. Furthermore, in 
making inferences from online data studies, Sudweeks and Simoff (1999) state that perhaps the biggest 
issue with CMC field research is in the near impossibility of replicating its studies. This daunting task, they 
remark, is currently being researched more thoroughly for furthering the validity of online evaluations. 

2.1 Evaluating online participation 

There is a common understanding now that online and offline activities and spaces cannot be divorced in the 
study of virtual community participation. However, how to go about evaluating such activities continues to 
pose as a formidable challenge. Kendall (1999) exemplifies some of the dominant challenges through her 
three-year experiences with Bluesky, a text only online forum. To measure interaction within this forum, she 
chose an eclectic range of methods: face-to-face interviews, participant observation online, participation in 
face-to-face gatherings, and reading of related newsgroups and email lists. In her analysis, she concludes 
that while it is understood that online interaction cannot be divorced from social and political offline contexts, 
the feasibility and expense issue in researching both domains is often insurmountable. She also warns us to 
not assume that there is a natural congruence between the offline-online worlds. In other words, we cannot 
look at online activities as extensions of offline activities, events and social and cultural norms and values. 

This thinking is echoed by Leander and McKim (2003) wherein the investigation of real-virtual spaces is 
rooted in the understanding of “the processes by which social spaces are held apart and blended, and how 
boundaries and blends are recognized in everyday practice” (p. 235). They envision though that over time, 
online communities will become more natural and familiar, blurring further the lines between these binaries. 
All the more reason to view the “place” for research online differently states Spradley (1980). Unlike offline 
traditional data collection at a “place,” he asks the readers to view online “place” instead as a “social 
situation” (p.19). The other two aspects that compose a sense of place in an online setting, he claims are its 
actors and activities where its physicality can be marked as a “basis for a social situation as long as the other 
two elements are also present” (p. 19). 

When it comes to examining the role of actors in computer-mediated communications (CMC), there is a 
special emphasis on the role of the researcher. This is a critical issue for many researchers when adapting 
the traditional ethnographic approach of participant observation to an online setting. For example, Kendall 
(2004) recommends participant observations for online interactive forums to prevent avoidance of important 
social contexts of online communication and interactions. However, such participation comes with its own 
challenges. Leander and McKim (2003) remarks on the researchers’ double bind, wherein when a 
researcher “observes” online, she is perceived by participants as a “lurker,” one who reads but does not post 
online (p.216). This is often looked upon negatively by the participants. Yet, if the researcher does not lurk 
but instead joins the online discussion, the problem shifts from lack of interaction to that of authenticity. 

On the other hand, Moss and Shank (2002) champion the role of the researcher as also that of a participant, 
stating that researchers from the “outside” cannot do justice to the understanding of data. The authors 
attribute the close readings of the data in their study to the fact that they were from the same learning 
communities. They express less concern on the issue of “objectivity” as they claim that they allowed the 
participants to speak for themselves. 
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3. Online pilot project on topic of “security” 

3.1 The CEBLE model 

The Critical Event-Based Learning Environment (CEBLE) was chosen for this pilot project as it is designed to 
help community facilitators think “past the application of their schematic responses” in response to recurrent 
problems as well as to capitalize on the multiple and unique perspectives of its participants (Lin, Schwartz & 
Hatano, 2005, p. 251). Originally designed for teachers, this model provides all participants the 

...opportunities to interact with people of different backgrounds, values, and goals through its multiple 
perspectives, and through sharing thoughts with other members of the communities. These social 
interactions allow teachers to gain the first-hand experience that the same problem can have many 
different solutions depending on the goals and values people bring with them, (p.251) 

The CEBLE shell evolved from a STAR Legacy software shell. The shell includes: (a) Meet the event (b) 
Generate responses to questions (c) Listen to multiple perspectives offered by people of different 
backgrounds, goals, and values (d) Act on the perspectives by generating solutions for selected perspectives 
(e) Reflect on the effectiveness of one’s solutions and share their choices of perspectives, solutions and 
legacies with other members in the community who explore a similar event and topic (p.251). 

This metacognitive thinking in the CEBLE model has been defined as “the awareness and regulation of the 
process of one’s thinking” and has been recognized as a critical ingredient to successful learning, online and 
offline (Lin, Schwartz & Hatano, 2005, p.4). In fact, this model promises potential within an online community 
environment given the popular belief that computer-mediated learning communities are based largely in 
situated cognition theory characterized by active engagement, discovery learning and co-construction of 
knowledge in communities of practice (Kirshner & Whitson, 1998; Lave & Wenger, 1991; Wenger, 1998). In 
particular, asynchronous communication is now commonly viewed as an effective tool for participant 
reflection and critical thinking (Luppicini, 2002). That said, this paper investigates this potential by applying 
this model to an online dialogue forum with participants from highly diverse, disadvantaged environments. 

3.1.1 Project goals 

This online pilot project was founded by an international non-profit in New York that uses dialogue for 
creating peace and understanding across boundaries. For the last fifteen years, they have facilitated 
dialogue within various settings and among diverse audiences. In the last year, they embraced the online 
medium as they sought to open this dialogue across international environments, particularly amongst actors 
that were marginalized socially, economically, politically and otherwise. The goals of this dialogue as 
professed and made explicit to the participants by the non-profit were mainly three fold: 

■ To generate a deeper understanding of the topic on “Security: How can we all have it?” amongst 
international participants 

■ To foster participation from a range of different cultures to broaden perspectives 

■ To apply cross-cultural awareness to critical problems participants face in their own communities 
to generate novel solutions to chronic problems related to security. 

The research goals however, for the scope of this paper is to gain insight to the following: 

1. Is there a change in “engagement” with a change in the instructional design model online? 

2. Does the CEBLE instructional design model enhance “engagement” within an online dialogue setting? 

3.1.2 Methodology 

Particular themes were drawn out and categorized from the discussion dataset. This heuristic coding 
(Jensen, 2002) produced a working document of elements and occurrences in the dataset. This analysis 
embraces the initial stages of the computer-mediated discourse analysis (CMDA) interpretive approach 
described by Herring (2004) starting with some of the situational and technological variables of the pilot 
project. Some of the situational variables entail the participant structure (number of participants), participant 
characteristics (background, motivation, experience, demographic etc), setting, purpose, topic, tone, 
linguistic code; some of the technological variables are synchronicity, anonymity, and channels of 
communication to name a few. 
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Also applied was the “manifest content coding reproduction” conceptual framework of Luppicini (2002) to 
determine the indicators of “engagement” online. In determining characteristics of this dialectic engagement, 
linkages were made to the goals of the non-profit for online dialogue: exposure to other perspectives and a 
deeper understanding of the topic “security” for generating ideas and solutions. In doing so, the following 
coding set was looked at as an aggregate of factors that constituted “engagement.” Some key codes are as 
follows: 

1. 1) Number of postings: The assumption is that the higher the number of postings, the more participation 
in the forum. 

2. 2) Participants referencing their own personal experiences: Dahlberg’s (2004) "inherent characteristics" 
of CMC was used, particularly the personalization/ depersonalization or intimacy as a determinant to 
measure engagement. The assumption here is that the personalization of a posting is indicative of “trust” 
and relationship building on the forum. 

3. 3) Participants referencing the topic: Here, if participants are not addressing the topic and are going off 
on a tangent, it can be perceived as disruptive and disengaging within the “flow” (Hine, 2000) of the 
dialogue. Hence, each posting that addresses the topic would be considered as that contributing to 
higher engagement. 

4. 4) Participants referencing each other: This is looked at as synonymous to “listening” within an online 
setting. The assumption here is that when a participant quotes or “refers” to a posting by another 
participant, it implies that they are reading the postings and building on it to further dialectic engagement. 

5. 5) Participants furthering dialogue: This category is intended to incorporate the aspect of higher-order 
thinking that is a sign of high engagement. When a participant posts a comment that is not regurgitating 
past dialogue but taking the past dialogue to a new and different level, it is considered an aspect that 
increases dialectic engagement. This is measured by looking at the prior postings and seeing if the 
posting at hand “adds” to the subject of security. Participants asking questions that compel participants to 
respond with new insights is also considered to have furthered the dialogue. 

Hence, each posting got a point for catering to each of these categories. The dataset was clumped into 
seven sessions, six that were traditional and the seventh that was the CEBLE model. Within each session, 
“engagement” was measured based on this coding structure and compared to one another and the CEBLE 
session to help determine whether there had been any change in dialectic engagement using the CEBLE 
model. In the CEBLE online model, the action research method was adopted to strengthen the validity of the 
study (Moss and Shank, 2002) wherein the author was on the “inside” of this online learning community 
playing the dual role of a facilitator and participant. However, the authors’ postings were not counted. 

3.1.3 Participants 

This organization employed the EARN platform, the largest international education resource network 1 to 
facilitate an online dialogue on “security”. In order to participate, each person had to be registered with 
EARN. Most participants were recruited primarily through outreach activities on EARN and idealist.org 
website, another popular non-profit Web Site. All participants had some access to computers and the 
internet as well as English proficiency skills. As we can see in Tablel, there were ten participants and one 
facilitator. The ratio of men to women on the forum is not overly skewed (4: 6). The participants are mainly 
social workers engaged in the field of migration, children and human rights, HIV/AIDS, drug prevention and 
the like. As we can see in Table 1, most have chosen to work in environments that are isolated, 
disadvantaged and marginalized in one form or another. 

The topic of “security” for this online forum was decided by the non-profit organization particularly in 
response to the 9/11 event. At the start, each participant introduced him/herself to all members of the 
organization. They were specifically asked to share with the group their reason/ motivation for participating in 
the dialogue. About half of the participants expressed overt interest in understanding people from 
international and diverse backgrounds. Furthermore, while two members (participant 2 & 3) expressed 
interest and motivation in attaining “peace” through dialogue, five of the members alluded to the subject of 
“rights” in addressing security issues. For example, participant # 4 from the Philippines wrote, "I am 
interested with this international conversation about security because my six years in working with children in 


1 Started in 1988, EARN is the world's largest non-profit global network that enables teachers and young 
people to use the Internet and other new technologies to collaborate on projects that are aimed to enhance 
learning. See http://www.iearn.org/ for more details. 
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extremely difficult situation in the Philippines, I realized the value of security issues specially in empowering 
the rights holders in claiming their rights" (Table 1). 

Table 1: Background and initial motivation of participants of online pilot project 


Participants 

Country 

Gender 

Affiliated field 

Stated reason for participation: QUOTES from Forum 

i 

NY, USA 

F 

Social work: Church 

"...after the US was attacked on 9/11/01 and this powerful 
country began focusing on its security...all of us in the world 
have security issues and security rights and we might grow to 
understand one another better in a conversation on this topic." 

2 

NY, USA 

F 

Social Work 
volunteer: Retired 

"1 want to explore what kind of societies we need to create in order to 
have peace." 

3 

Zambia 

M 

Social work: HIV/ 
AIDS 

"1 believe that all the wars and rumours of war that we have see 
and heard respectively are as a result of failure by people to 
come to a round table and amicably discuss and resolve 
differences...Getting views from different backgrounds may in its 
own way add something to the search for global peace which to 
me is not beyond reach." 

4 

Philippines 

F 

Social work: Child 
labor 

"1 am interested with this international conversation about 
security because my six years in working with children in 
extremely difficult situation in the Philippines, 1 realized the 
value of security issues specially in empowering the rights 
holders in claiming their rights." 

5 

Zimbabwae/ 
South African 

F 

Social Work 

"Security is an important aspect in our lives and the more we can 
debate the various issues relating to it the better we can find 
answers." 

6 

LA, USA 

F 

Shelters 

"We feel like it our duty to make sure that everyone in our own 
community know that they have the right to feel secure while it is 
our responsibilty promote security within our own communities 
and to hold our governments repsonible as well." 

7 

Turkey 

M 

Student 

Cultural studies interest 

8 

Zurich, 

Switzerland 

M 

Social work: Child 
Abuse 

"...working with victims of abuse and detention is pushing my 
motivation even harder" 

9 

Philippines 

M 

Social work: Elderly 
Care 

"...our long range goal is to have a nation-wide organization of older 
people (mostly poor, mostly women) who will ensure real security." 

10 

NY, USA 

F 

Social work: Drug 
prevention 

"1 am happy to learn about the experiences and lives of people 
living so differently from me." 


NY USA/ 
Caucasian 

NY, USA/ 
Indian 

F 

F 

Traditional model/ 
Masters student 
CEBLE model 
facilitator/ Doctoral 
student 

1 am doing my doctoral studies in technology, literacy and 
language and 1 want to understand what makes for effective and 
meaningful dialogue online. 


3.1.4 Application of the CEBLE model 

In terms of sessions, there were a total of seven sessions in the online forum: six using the “traditional” 
instructional model and one using the CEBLE instructional model (see Table 2). The instructional model in 
the past six sessions is termed here as the “traditional” model for comparative sake only. The impetus to use 
a new model by the non-profit was based on the intent to increase the quantity and quality of postings in the 
online forum. To address this challenge, the CEBLE model was applied to the seventh session of the forum. 
Given that the online forum was a dynamic international and cross-cultural environment with a goal for critical 
thinking on the issue of security, it was hypothesized that the CEBLE model may be appropriate to foster 
dialogue, particularly by generating questions and facilitating discussion grounded within the context of past 
participant dialogue and exposure to new viewpoints for higher engagement (see Table 2). As we can see 
below, the instructional strategies between the two models to create an engaging online dialogue were 
different. 
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Table 2: Comparing the traditional and online CEBLE model 


STEPS 

Traditional model (n = 10) 

Online CEBLE model (n = 10) 

At the beginning 
(Meet the Event) 

A question is posed mostly 
independent of references to past 
dialogue 

A question is situated in past dialogue 

External experts’ 
view (Multiple 
perspectives) 

Not offered 

Offered in the middle of the discussion as a 
contradictory viewpoint to participant posting. 

Facilitation: 
facilitator’s role 
(Generate ideas 
and responses) 

Posts questions / Affirms 
responses 

Posts questions / Affirms responses/ 
Provides alternate viewpoints/dual role of 

participant 


As we can see in Table 2, to initiate discussion in the traditional model, the facilitator posed a question to the 
participants as a separate discussion link for each of the six sessions. This question was posted as an 
independent entity without referencing past participant dialogue. When participants responded to the 
question, the facilitator primarily affirmed the participants’ responses. On the other hand, in the CEBLE 
model, the question was rooted in past dialogue to determine trends of consensus and dissent. The question 
was then built around this critical synthesis to further the understanding on security (see Abstract 1) 

Abstract 1 : Meet the event: Initiation of dialogue through question in the traditional and CEBLE model 

12:45pm Sept 19, 2005 EDT 

What are the biggest obstacles to secirity in you locality? 

03:47pm llov 1, 2005 EDT 


Hi everyone, 

At this stage in our dialo gue on secir ity, it seems that there is an underlying dilemma on the power of the individual in creating secirity. 
From my understanding, seems more biased on individuals having more power and control over their personal secirity. She 

parallels safety to security -referring to "wellbeing without fear" and states that participation and intervention is almost negligible as we 
reach the national a/and international level. Interestingly, | refers to security as that which needs to be built, where '^security can be 
gained not so much from physical or armed strength but on mutual respect." Here, I would interpret this as security that is communal in 
nature. In theory of "lovin g conflict," an interesting thought has surfaced where the oppressed seems to have more power than the 
oppre ssors. In the same vein, suggests that dialogue is the best tool to sort out conflict than any other means of force while 

m§| shares his view on conflict which is not necessarily negative as long as we knew how to negotiate with it. 

So it seems to me that we all seem to agree at this stage that secirity has multiple hues and is layered as individual, communal, social, 
economic, national, international and so on. We also seem to see a relationship between conflict and secirity that is not necessarily 
diametrically opposite to one another. However, I sense that as we speak about secirity, we have not made our beliefs explicit on how 


we negotiate with 

power 

to 

achieve secirity for 

ourselves and others. 

So my 

question is as 

follows: 

In what way 

have 

you 

exercised yoir 

power to achieve 

security 

for yourself/ 

others? 


Looking forward to hearing yoir views on this. 

The other distinguishing aspect between the two models is in the strategic posting of an outside and 
provocative perspective through the CEBLE model (Table 2). This feature was absent in the traditional 
model. This aspect was incorporated to supply a contradictory and perhaps unfamiliar viewpoint to the 
participants to make them reevaluate their perspectives and engage at a higher metacognitive level. As we 
can see in Abstract 2, there is back and forth dialogue between the participant and the facilitator. Prior to 
this, there was a general consensus amongst the participants on the idea that migration produced insecurity 
as exemplified in the first posting in Abstract 2. The outside (expert) perspective (Second posting: Abstract 2) 
introduced by the facilitator was deliberately chosen to contradict this perception. The posting was on the 
“urban tribes” theory proposed by Ethan Wattars stating that migration can cause a reverse effect, where the 
need for security amongst the displaced populace can create tighter communities. 


www.ejel.org 


6 


©Academic Conferences Ltd 




Payal Arora 


Abstract 2: Sampling of CEBLE dialogue before and after outside perspective is introduced 


|llov 17, 2005 6:50 an (5.) 
(IP: 128.59.103.1381 


Bookirak | Email to Rientl | Edit | Delete | Reply |Move 


Dear (■! 

Your wwi personal stay about "security" could be repeated millions of times by Filipinos. No country in the world, percentage wise has 
as many people seeking economic security abroad. Yesterdays broadsheet had it that for the frst ten months of this year, 800,000 
filipinos have gone abroad seeking employment (the official figure) which means a million people-mostly young this year. Altogether, 
maybe 10 million or 15% of the population at any given time (this does not include the illegals who are also estimated to be large!) We 
have a close friend who spert 13 years in Saudi as a driver. The good part: his four children are all college educated. The bad part: they 
grew up without a father and hardly knew him. Its a trade-off many have to make. This does not include all the rackets, exploitation and 
sex trade to which a number, in their desperation, have fallen into. Now that's insecurity! 
peace 


|Hov 19, 2005 2:07 |Mii <6.> 

Bookmak | Email to RieiKl | Edit | Delete | Reply |Move 

Dear 

... I wonder if immigration can have a reverse effect?... Interestingly, I came across a book called “Urban Tribes" by Ethan Wattars that 
talks of the urban jungle of cities. He speaks of the proactiveness of isolated people in reviving the tribal ways of strong expectations 
from their 

friendships, and sharing and caring for eachother much like a traditional tribe... 

^^■1 - Nov 21,2005 5:05 pm (7.) 

Bookmark | Email to Friend | Edit | Delete | Reply | Move 
Dear 

I was at a book launching the other day. a book about disabled people who had famed themselves into cooperatives and had 
metamophesed from begging in the streets to manufacturing chairs fa elementary schools. At the launching a good friend spoke 
saying, "we ere all disabled in one way or anotha and you have taught us how to deal with our own disability "..change disability to 
insecurity and the same truth may apply. In other wads, its not two questions but only one...my security and everyone eises are 
interdependent ...and my security will depend on my engagement in everyone eises... 


3.1.5 Results from the CEBLE application: 
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Figure 1: Comparison of traditional model and CEBLE model: Indicators for engagement 

As we can see in Figure 1 above, there is some change with the change in the instructional design model. If 
we were to discount session 1 when comparing the indicators, the CEBLE session can be seen as showing 
positive results over the traditional sessions (Session 1 to Session 6). Also, the postings in the CEBLE 
session marginally increased over all sessions but for session 1. However, the difference appears less 
significant. While there is a total of seven postings in the CEBLE session, this was still less than the total 
number of participants (n =10). On the other hand, there seemed to be more significant changes in the 
content of postings in the CEBLE session. For example, while most participants referred to the topic across 
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sessions, be it the CEBLE or traditional sessions, there had been a substantive increase in the 
personalization aspect within the CEBLE model. In other words, participants appeared to draw much more 
from their own experiences when responding within the CEBLE model. They seemed to share perspectives 
from their own contextual environment when explaining their point of view. 


The only other session that competed on this personalization aspect is session 1. In fact, as we can see in 
Figure 1, there are a number of similarities between session 1 of the traditional model and session 7 from the 
CEBLE model. They both had high number of postings, high referencing to the topic and more importantly, 
they both scored high on the personal reference category. Six of the nine participants in session 1 weaved in 
their own experiences when responding to the question. In the CEBLE model, five of the seven participants 
did the same. Also, few participants in session 1 referred to one another or furthered the topic with questions 
or insights on the topic of security. In terms of furthering the dialogue, there was a gradual increase in 
participants demonstrating critical comments or questions that “furthered” the concept of security. Given that 
this is a harder category to make explicit, here is a sample of what constituted as “furthering the discussion.” 


Oct 14, 2005 9:14 am (3.) 


You cant have security because you want security. You have to think, what makes for security? (your 
question) because it is consequent or contingent on something else. Some people seem to think it’s 
strength—gotta show you’re strong.’’the best defense is a good offence” but security is built not so 
much on physical or armed strength but on mutual respect —a kind of strength in itself. In our case, it 
means going door to door and spending months talking to people until we are accepted and can get 
everyone to sit down together and talk about what they would like to do together—as a community (do 
you want better health care—are you willing to be trained as community health workers? Etc.) now, we 
can talk about security—not because we set out to get more security but because we share something 
of the same values, the same dreams...now we are secure because we are in this together... 

And here’s a sample of what did not constitute as furthering the discussion... 

I am under stress wi th a new job which involves a lot of travel. I will respond on later for a while. And I 
wanted to say that ,n Portugal has results of a discussion, but has been having some 

difficulty getting it translated to English. 


However, if we were to not discount Session 1, the results are less definitive. Overall, to answer the research 
questions on whether there were any changes whatsoever in the change in instructional design as well as to 
see if the CEBLE model enhanced engagement, results can be found to be affirmative on both accounts only 
if accounting for Session 1. This in itself is not sufficient to profess that CEBLE does in fact increase 
engagement as there are some confounding factors as we see in the analysis section below besides the 
more obvious small sample size that deters generalization. The purpose of this paper though lies in 
explicating the evaluation strategies and challenges involved in online dialogue measurement as well as 
gauging the potential of the CEBLE model for engagement. 


3.1.6 Analysis of findings 

The reason we can consider discounting Session 1 is due to it being the first session where participants 
have to address the topic as they are explicitly asked to do so by the facilitator. This can explain the high 
number of postings, personal references and reference to the topic. To the extent that in session 1, the 
discussion was at its initial stage where the participants were getting to “know” one another and was starting 
to “build a relationship” with the group, it can be seen as an anomaly. Further, as is commonly experienced 
in voluntary participation, there is generally an enthusiasm at the start of an endeavor. The challenge for 
most online discussion boards is in sustaining and enhancing participation through engagement and not just 
the recruitment of the participants. 

Also, if we take a look at the questions in session 1, the reason for the personalization aspect becomes more 
evident. The question being: “When you were growing up, what did you learn about how to resolve 
arguments or other conflicts? What do you think now?” This was deliberately structured to evoke 
personalized responses. In contrast, the question for the CEBLE session (Abstract 1) was framed as a 
synthesis of participant’s past comments to further the dialogue to new insights. With that logic in mind 
however, the tables can be reversed. In effect, the CEBEL session personalization can also be seen as 
"understandable" because the question in that session was posed to maximize that aspect. Of course, this 
brings into question whether or not its’ the CEBEL model that is causing the result, or simply the type of 
question. 
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That said, even if we do discount Session 1, the increase in the postings seems unimpressive and cautions 
against clear declarations on engagement levels. While this in itself is worth pointing to, what is of more 
importance here is that this category fails to include “lurkers” which is critical to the understanding of 
engagement. In theory, a participant could never post a comment online and yet be highly “engaged” by 
reading all the postings and reflecting upon it in their offline lives. The usage of log files of the online forum 
was considered to determine whether participants were viewing the comments. However, it was discovered 
that users do not necessarily have to visit the forum in order to actively participate in the topic. This is 
because every posting on the forum gets automatically sent as an email notification to the participants email 
account. Thereby, the most active participant theoretically could never have logged into the forum. Since 
many users had this option enabled, the log files would have been an unreliable measure of participation. 
Thereby, we have an unresolved confound between participant postings and engagement. 

Also, while having created categories or codes as a composite of “engagement,” it is questionable whether 
all codes are of equal weight in the gauging of engagement. Further, in relation to CEBLE’s multiple 
perspectives, it is worth asking whether simple disagreement automatically constitutes as a multiple 
perspective and whether multiple perspectives could in fact equate to engagement by itself. As to the 
researcher being the facilitator, there is always the research bias looming over the study that takes away 
from the results to a certain extent. At best, a compelling case can be made that CEBEL came out with some 
potential in generating personalized engagement. 

4. Conclusion 

This study was particularly fortunate to have the same number of participants across all sessions. This 
fostered stronger validity within the pilot project. Being asynchronous and by invite only, this online pilot 
study was able to circumvent some of the challenges that incur within most synchronous settings such as 
participant anonymity, changing online demographics within the time of the study, etc. Also, the controlled 
setting with separate discussion threads and archives for each session enabled the sorting of data. 

That said, given the marginal increase in the number of postings in the CEBLE model, the issue of 
increasing the quantity of postings still persists. Part of this is because of the small sample size. The log files 
dilemma as explicated in the reading of the postings remains unresolved. Other variables that could be 
influencing this data also need to be investigated. Given that the adult group specifically chosen by the non¬ 
profit represents an international demographic situated within marginalized environments, variables 
pertaining to access to technology and other related issues of access and usage need to be considered. 
Unique socio-cultural, economic and geographic factors such as lack of mobility to cybercafes, electricity, 
individual routine, political and social stability and the like could be impacting the amount and quality of 
postings by the participants. This reinforces the inseparable nature between offline and online lives. 

Also, while attempting to execute a fair comparison between the two models, the discrepancy of session 
length is evident. While the traditional model went on for approximately six months, the CEBLE model was 
executed for a period of one month only. Hence, a more longitudinal approach is needed to gauge impact. 
While the number of postings is not significant, the observed difference between the nature of postings 
between the two models is encouraging. Based on these preliminary findings, there seems to be some 
support for the CEBLE model as a potential engagement tool in instructional design within online dialogue. 

For future studies, it would also be useful to know what aspects of the CEBLE model are most effective in 
online dialogue. To conclude, it is important to remember that in evaluating socially disparate sub-cultural 
group participants, we should not presume that all online interaction occurs within the same homogenous 
cultural context separate from the different cultures of the participants who contribute to it and that 
“engagement” means different things to different individuals from different socio-cultural backgrounds. 
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